{
  "cells": [
    {
      "metadata": {
        "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
        "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
        "trusted": true
      },
      "cell_type": "markdown",
      "source": " #  <div style=\"text-align: center\">  Santander ML Explainability  </div> \n###  <div style=\"text-align: center\">CLEAR DATA. MADE MODEL. </div> \n<img src='https://galeria.bankier.pl/p/b/5/215103d7ace468-645-387-261-168-1786-1072.jpg' width=600 height=600>\n<div style=\"text-align:center\"> last update: <b> 20/02/2019</b></div>\n\n\n\nYou can Fork code  and  Follow me on:\n\n> ###### [ GitHub](https://github.com/mjbahmani/10-steps-to-become-a-data-scientist)\n> ###### [Kaggle](https://www.kaggle.com/mjbahmani/)\n-------------------------------------------------------------------------------------------------------------\n <b>I hope you find this kernel helpful and some <font color='red'>UPVOTES</font> would be very much appreciated.</b>\n    \n -----------"
    },
    {
      "metadata": {
        "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
        "collapsed": true,
        "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a",
        "trusted": false
      },
      "cell_type": "markdown",
      "source": " <a id=\"top\"></a> <br>\n## Notebook  Content\n1. [Introduction](#1)\n1. [Load packages](#2)\n    1. [import](21)\n    1. [Setup](22)\n    1. [Version](23)\n1. [Problem Definition](#3)\n    1. [Problem Feature](#31)\n    1. [Aim](#32)\n    1. [Variables](#33)\n    1. [Evaluation](#34)\n1. [Exploratory Data Analysis(EDA)](#4)\n    1. [Data Collection](#41)\n    1. [Visualization](#42)\n    1. [Data Preprocessing](#43)\n1. [Machine Learning Explainability for Santander](#5)\n    1. [Permutation Importance](#51)\n    1. [How to calculate and show importances?](#52)\n    1. [What can be inferred from the above?](#53)\n    1. [Partial Dependence Plots](#54)\n1. [Model Development](#6)\n    1. [lightgbm](#61)\n    1. [RandomForestClassifier](#62)\n    1. [DecisionTreeClassifier](#63)\n    1. [CatBoostClassifier](#64)\n    1. [Funny Combine](#65)\n1. [References](#7)"
    },
    {
      "metadata": {
        "_uuid": "af63d485947264b68c37768c73ffac21bb631c0f"
      },
      "cell_type": "markdown",
      "source": " <a id=\"1\"></a> <br>\n## 1- Introduction\nAt [Santander](https://www.santanderbank.com) their mission is to help people and businesses prosper. they are always looking for ways to help our customers understand their financial health and identify which products and services might help them achieve their monetary goals.\n<img src='https://www.smava.de/kredit/wp-content/uploads/2015/12/santander-bank.png' width=400 height=400>\n\nIn this kernel we are going to create a **Machine Learning Explainability** for **Santander** based this perfect [course](https://www.kaggle.com/learn/machine-learning-explainability) in kaggle.\n><font color=\"red\"><b>Note: </b></font>\nhow to extract **insights** from models?"
    },
    {
      "metadata": {
        "_uuid": "477c78003f7e8a9fa7c83a2e61950f316cc2d26f"
      },
      "cell_type": "markdown",
      "source": "<a id=\"2\"></a> <br>\n## 2- A Data Science Workflow for Santander \nOf course, the same solution can not be provided for all problems, so the best way is to create a **general framework** and adapt it to new problem.\n\n**You can see my workflow in the below image** :\n\n <img src=\"http://s8.picofile.com/file/8342707700/workflow2.png\"  />\n\n**You should feel free\tto\tadjust \tthis\tchecklist \tto\tyour needs**\n###### [Go to top](#top)"
    },
    {
      "metadata": {
        "_uuid": "733f1f0e45933f46a674df4d9ee6561de156d748"
      },
      "cell_type": "markdown",
      "source": " <a id=\"2\"></a> <br>\n ## 2- Load packages\n  <a id=\"21\"></a> <br>\n## 2-1 Import"
    },
    {
      "metadata": {
        "_kg_hide-input": true,
        "trusted": true,
        "_uuid": "7646c73810d475601436c096d36498cfaa489ec4"
      },
      "cell_type": "code",
      "source": "from sklearn.model_selection import train_test_split\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.tree import DecisionTreeClassifier\nfrom catboost import CatBoostClassifier,Pool\nfrom IPython.display import display\nimport matplotlib.patches as patch\nimport matplotlib.pyplot as plt\nfrom sklearn.svm import NuSVR\nfrom scipy.stats import norm\nfrom sklearn import svm\nimport lightgbm as lgb\nimport xgboost as xgb\nimport seaborn as sns\nimport pandas as pd\nimport numpy as np\nimport warnings\nimport time\nimport glob\nimport sys\nimport os\nimport gc",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "ec9e63f88e677ed60bfe5a8de4bb0b2a8be8b00a"
      },
      "cell_type": "markdown",
      "source": " <a id=\"22\"></a> <br>\n##  2-2 Setup"
    },
    {
      "metadata": {
        "_kg_hide-input": true,
        "trusted": true,
        "_uuid": "16768a965c3ced6a76d33642e11ecae18f5977e8"
      },
      "cell_type": "code",
      "source": "# for get better result chage fold_n to 5\nfold_n=2\nfolds = StratifiedKFold(n_splits=fold_n, shuffle=True, random_state=10)\n%matplotlib inline\n%precision 4\nwarnings.filterwarnings('ignore')\nplt.style.use('ggplot')\nnp.set_printoptions(suppress=True)\npd.set_option(\"display.precision\", 15)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "9bcec14c7403174f2a9fdab866ea8248aa76328d"
      },
      "cell_type": "markdown",
      "source": " <a id=\"23\"></a> <br>\n## 2-3 Version\n"
    },
    {
      "metadata": {
        "_kg_hide-input": true,
        "trusted": true,
        "_uuid": "4644581a5ce6c2c98ec2668a95037742b189c318"
      },
      "cell_type": "code",
      "source": "print('pandas: {}'.format(pd.__version__))\nprint('numpy: {}'.format(np.__version__))\nprint('Python: {}'.format(sys.version))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "ccfe7d54dac0cf31125a49b1981d25a69e679db0"
      },
      "cell_type": "markdown",
      "source": "<a id=\"3\"></a> \n<br>\n## 3- Problem Definition\nIn this **challenge**, we should help this **bank**  identify which **customers** will make a **specific transaction** in the future, irrespective of the amount of money transacted. The data provided for this competition has the same structure as the real data we have available to solve this **problem**.\n"
    },
    {
      "metadata": {
        "_uuid": "dc84d5445dab932c46e2c5dfe86f2131e98aa0f1"
      },
      "cell_type": "markdown",
      "source": "<a id=\"31\"></a> \n### 3-1 Problem Feature\n\n1. train.csv - the training set.\n1. test.csv - the test set. The test set contains some rows which are not included in scoring.\n1. sample_submission.csv - a sample submission file in the correct format.\n"
    },
    {
      "metadata": {
        "_uuid": "449535074de09edf5cdccb5a63539661fff29be6"
      },
      "cell_type": "markdown",
      "source": "<a id=\"32\"></a> \n### 3-2 Aim\nIn this competition, The task is to predict the value of **target** column in the test set."
    },
    {
      "metadata": {
        "_uuid": "184d1004ff1f90fdf9d475cbb37ea94b89dd2567"
      },
      "cell_type": "markdown",
      "source": "<a id=\"33\"></a> \n### 3-3 Variables\n\nWe are provided with an **anonymized dataset containing numeric feature variables**, the binary **target** column, and a string **ID_code** column.\n\nThe task is to predict the value of **target column** in the test set.\n"
    },
    {
      "metadata": {
        "_uuid": "81b77e49bae78bad2c31bc2ce101c1fff8296af0"
      },
      "cell_type": "markdown",
      "source": "<a id=\"34\"></a> \n## 3-4 evaluation\n**Submissions** are evaluated on area under the [ROC curve](http://en.wikipedia.org/wiki/Receiver_operating_characteristic) between the predicted probability and the observed target.\n<img src='https://upload.wikimedia.org/wikipedia/commons/6/6b/Roccurves.png' width=300 height=300>"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "9ba4b5a56a9c67899ed54c7e4dad0930ce90fbfa"
      },
      "cell_type": "code",
      "source": "from sklearn.metrics import roc_auc_score, roc_curve",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "f8a54abda5d2e30c5cdb25c554bef1f8a341482c"
      },
      "cell_type": "markdown",
      "source": "<a id=\"4\"></a> \n## 4- Exploratory Data Analysis(EDA)\n In this section, we'll analysis how to use graphical and numerical techniques to begin uncovering the structure of your data. \n*  Data Collection\n*  Visualization\n*  Data Preprocessing\n*  Data Cleaning\n<img src=\"http://s9.picofile.com/file/8338476134/EDA.png\" width=400 height=400>"
    },
    {
      "metadata": {
        "_uuid": "c90af9d21a49adcbd478c56871149f1282c58b7f"
      },
      "cell_type": "markdown",
      "source": " <a id=\"41\"></a> <br>\n## 4-1 Data Collection"
    },
    {
      "metadata": {
        "_kg_hide-input": true,
        "trusted": true,
        "_uuid": "1c883e09dbe7e4b4d7caeb2cf380ec0f07209531"
      },
      "cell_type": "code",
      "source": "print(os.listdir(\"../input/\"))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_kg_hide-input": true,
        "trusted": true,
        "_uuid": "7427d2d9ce7dff9d63a355631a1718a6fb6c5847"
      },
      "cell_type": "code",
      "source": "# import Dataset to play with it\ntrain= pd.read_csv(\"../input/train.csv\")\ntest = pd.read_csv('../input/test.csv')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_kg_hide-input": true,
        "trusted": true,
        "_uuid": "9be60393f64121beb23124bdec3cef638436e788"
      },
      "cell_type": "code",
      "source": "sample_submission = pd.read_csv('../input/sample_submission.csv')\nsample_submission.head()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "6acf9a55418407ab1da0e7a8c4e1f540a2618ac3"
      },
      "cell_type": "code",
      "source": "train.shape, test.shape, sample_submission.shape",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_kg_hide-input": true,
        "_uuid": "1d054476a562f14b8d7e88fd2a8ea3b7b70c371c"
      },
      "cell_type": "code",
      "source": "train.head(5)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "3820345bbd08346359e699ead3f125f438dbaefc"
      },
      "cell_type": "markdown",
      "source": " <a id=\"41\"></a> <br>\n##   4-1-1Data set fields"
    },
    {
      "metadata": {
        "_kg_hide-output": true,
        "trusted": true,
        "_uuid": "6e5f5d0a8e6057a19c41d0698c76cfa5f0e61202"
      },
      "cell_type": "code",
      "source": "train.columns",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "1d31392a34a8bba87565f4a00a44084641a28f5b"
      },
      "cell_type": "code",
      "source": "print(len(train.columns))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "2b8083e0022c342963869e609a65bae49cbe2d2a"
      },
      "cell_type": "code",
      "source": "print(train.info())",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "1292b57ec3e35831def62fa0a51aebac73df3b69"
      },
      "cell_type": "markdown",
      "source": " <a id=\"422\"></a> <br>\n## 4-2-2 numerical values Describe"
    },
    {
      "metadata": {
        "_kg_hide-input": true,
        "trusted": true,
        "_uuid": "ca9ae169e960fddc581b8da1b15c7e79d8bbe972"
      },
      "cell_type": "code",
      "source": "train.describe()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "c1eddbf9a17cb7b15b342b4d2d48af7f0ac79006"
      },
      "cell_type": "markdown",
      "source": " <a id=\"42\"></a> <br>\n## 4-2 Visualization"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "70207522d405040466649db4664f6dc9cb75849f"
      },
      "cell_type": "markdown",
      "source": "<a id=\"421\"></a> \n### 4-2-1 hist"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "51042af4ef472a7863f9698fdca2d99eefa0af7d"
      },
      "cell_type": "code",
      "source": "train['target'].value_counts().plot.bar();",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "0cf04a9128e231d216018ba2c23244d6c041e209"
      },
      "cell_type": "code",
      "source": "f,ax=plt.subplots(1,2,figsize=(20,10))\ntrain[train['target']==0].var_0.plot.hist(ax=ax[0],bins=20,edgecolor='black',color='red')\nax[0].set_title('target= 0')\nx1=list(range(0,85,5))\nax[0].set_xticks(x1)\ntrain[train['target']==1].var_0.plot.hist(ax=ax[1],color='green',bins=20,edgecolor='black')\nax[1].set_title('target= 1')\nx2=list(range(0,85,5))\nax[1].set_xticks(x2)\nplt.show()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "225972f02d20b1aed1a2d6a3dae7e2c14b7caad1"
      },
      "cell_type": "markdown",
      "source": " <a id=\"422\"></a> <br>\n## 4-2-2 Mean Frequency"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "427a904f784fbf353c728e3abafa8628ffa0a52c"
      },
      "cell_type": "code",
      "source": "train[train.columns[2:]].mean().plot('hist');plt.title('Mean Frequency');",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "1498698d6c5175ea7a10ff3595d9f0177e1e6861"
      },
      "cell_type": "markdown",
      "source": "<a id=\"423\"></a> \n### 4-2-3 countplot"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "90cae6c076bcefc862828b177672c2e481b6a15b",
        "_kg_hide-input": true
      },
      "cell_type": "code",
      "source": "f,ax=plt.subplots(1,2,figsize=(18,8))\ntrain['target'].value_counts().plot.pie(explode=[0,0.1],autopct='%1.1f%%',ax=ax[0],shadow=True)\nax[0].set_title('target')\nax[0].set_ylabel('')\nsns.countplot('target',data=train,ax=ax[1])\nax[1].set_title('target')\nplt.show()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "565eea91ba475dfb935f89052738e98c4338206a"
      },
      "cell_type": "markdown",
      "source": "<a id=\"424\"></a> \n### 4-2-4 hist\nif you check histogram for all feature, you will find that most of them are so similar"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "4cafe1f825c8f5b7d1115411cf6018649ae2c2a1",
        "_kg_hide-input": true
      },
      "cell_type": "code",
      "source": "train[\"var_0\"].hist();",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "291bf70bbea5066cf44df3e4a1093d8b95999910"
      },
      "cell_type": "code",
      "source": "train[\"var_81\"].hist();",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "4fca943180a87f02bf9cdb0802835de1c1114df0"
      },
      "cell_type": "code",
      "source": "train[\"var_2\"].hist();",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "c7c539dd31e2d389813645c30e997f1728e9f926"
      },
      "cell_type": "markdown",
      "source": "<a id=\"426\"></a> \n### 4-2-6 distplot\n the target in data set is **imbalance**"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "6b04bc254eee5dcdeac96c17f9c80a0657515a9b"
      },
      "cell_type": "code",
      "source": "sns.set(rc={'figure.figsize':(9,7)})\nsns.distplot(train['target']);",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "b710f6cb8eb852a3b749774172c911de598b986c"
      },
      "cell_type": "markdown",
      "source": "<a id=\"427\"></a> \n### 4-2-7 violinplot"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "b01885275f75488166c034eafe8a0ca9870ab70b"
      },
      "cell_type": "code",
      "source": "sns.violinplot(data=train,x=\"target\", y=\"var_0\")",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "c9c01ab969d0e85f4ba301bdb4656b8a6ec99d32"
      },
      "cell_type": "code",
      "source": "sns.violinplot(data=train,x=\"target\", y=\"var_81\")",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "8a436a7a7f1391cb78ae3cec6c8d5adb42f41bd2"
      },
      "cell_type": "raw",
      "source": ""
    },
    {
      "metadata": {
        "_uuid": "8afaaae9f83971dd442a16af434d88318d82c352"
      },
      "cell_type": "markdown",
      "source": " <a id=\"43\"></a> <br>\n## 4-3 Data Preprocessing\nBefore we start this section let me intrduce you, some other compitation that they were similar to this:\n\n1. https://www.kaggle.com/artgor/how-to-not-overfit\n1. https://www.kaggle.com/c/home-credit-default-risk\n1. https://www.kaggle.com/c/porto-seguro-safe-driver-prediction"
    },
    {
      "metadata": {
        "_uuid": "a03a95410ad45a7e176ca72a7c03d1b05d49eda6"
      },
      "cell_type": "markdown",
      "source": " <a id=\"431\"></a> <br>\n## 4-3-1 Check missing data for test & train"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "bbcb30c8b016481a47e46d02ef24241f17412027"
      },
      "cell_type": "code",
      "source": "def check_missing_data(df):\n    flag=df.isna().sum().any()\n    if flag==True:\n        total = df.isnull().sum()\n        percent = (df.isnull().sum())/(df.isnull().count()*100)\n        output = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\n        data_type = []\n        # written by MJ Bahmani\n        for col in df.columns:\n            dtype = str(df[col].dtype)\n            data_type.append(dtype)\n        output['Types'] = data_type\n        return(np.transpose(output))\n    else:\n        return(False)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_kg_hide-input": false,
        "_uuid": "6adc1fb6f7ef818946843b3698583ade2458a218",
        "_kg_hide-output": true
      },
      "cell_type": "code",
      "source": "check_missing_data(train)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "c7ac1823585acaa773a4507bcf4db9e3c0711bfd"
      },
      "cell_type": "code",
      "source": "check_missing_data(test)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "9dd09f6e2a06e6902257497eccbf24bec4bcde48"
      },
      "cell_type": "markdown",
      "source": " <a id=\"432\"></a> <br>\n## 4-3-2 Binary Classification"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "5f7f7d7af58eb891481afd42022ce5fb7e75ebbb"
      },
      "cell_type": "code",
      "source": "train['target'].unique()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "f9031786ba1022e0970303451664fbacd9687710"
      },
      "cell_type": "markdown",
      "source": " <a id=\"433\"></a> <br>\n## 4-3-3 Is data set imbalance?"
    },
    {
      "metadata": {
        "_uuid": "536e31bea4523d2a9bfdbc3ac8fd2518d3e2e2cb"
      },
      "cell_type": "markdown",
      "source": "A large part of the data is unbalanced, but **how can we  solve it?**"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "982afe00bd42859191af65e0fd43e06e74dbfab3"
      },
      "cell_type": "code",
      "source": "train['target'].value_counts()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "d89ebf7be40381c2aca9f78b79ddf11d291b423c"
      },
      "cell_type": "code",
      "source": "def check_balance(df,target):\n    check=[]\n    # written by MJ Bahmani for binary target\n    print('size of data is:',df.shape[0] )\n    for i in [0,1]:\n        print('for target  {} ='.format(i))\n        print(df[target].value_counts()[i]/df.shape[0]*100,'%')\n    ",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "2993992933bd7365e867d38fc619e17b2d6832e3"
      },
      "cell_type": "markdown",
      "source": "1. **Imbalanced dataset** is relevant primarily in the context of supervised machine learning involving two or more classes. \n\n1. **Imbalance** means that the number of data points available for different the classes is different\n\n<img src='https://www.datascience.com/hs-fs/hubfs/imbdata.png?t=1542328336307&width=487&name=imbdata.png'>\n[Image source](http://api.ning.com/files/vvHEZw33BGqEUW8aBYm4epYJWOfSeUBPVQAsgz7aWaNe0pmDBsjgggBxsyq*8VU1FdBshuTDdL2-bp2ALs0E-0kpCV5kVdwu/imbdata.png)"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "bda7929cf124ed55c06e57e6119ec3a1dff2b220"
      },
      "cell_type": "code",
      "source": "check_balance(train,'target')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "ad216b95bcbcf07e0b6ee6000f974acd0d2409b6"
      },
      "cell_type": "markdown",
      "source": "## 4-3-4 skewness and kurtosis"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "b669c23c1de584988629ac99bbbe0dd74a56dd7c"
      },
      "cell_type": "code",
      "source": "#skewness and kurtosis\nprint(\"Skewness: %f\" % train['target'].skew())\nprint(\"Kurtosis: %f\" % train['target'].kurt())",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "a1cba663f360dbbc2fe5a9d76ab219f153921922"
      },
      "cell_type": "markdown",
      "source": " <a id=\"5\"></a> <br>\n# 5- Machine Learning Explainability for Santander\nIn this section, I want to try extract insights from models with the help of this excellent [**Course**](https://www.kaggle.com/learn/machine-learning-explainability) in Kaggle.\nThe Goal behind of ML Explainability for Santander is:\n1. All features are senseless named.(var_1, var2,...) but certainly the importance of each one is different!\n1. Extract insights from models.\n1. Find the most inmortant feature in models.\n1. Affect of each feature on the model's predictions.\n<img src='http://s8.picofile.com/file/8353215168/ML_Explain.png'>\n\nAs you can see from the above, we will refer to three important and practical concepts in this section and try to explain each of them in detail."
    },
    {
      "metadata": {
        "_uuid": "2569c256bf9b387c2c0e7c06c473f0e841b6db7d"
      },
      "cell_type": "markdown",
      "source": " <a id=\"51\"></a> <br>\n## 5-1 Permutation Importance\n In this section we will answer following question:\n 1. What features have the biggest impact on predictions?\n 1. how to extract insights from models?"
    },
    {
      "metadata": {
        "_uuid": "45cb8e4392def434e0c7142f33a847837207dced"
      },
      "cell_type": "markdown",
      "source": "### Prepare our data for our model"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "9fafaf9cadb57f7c4d51489df141aa3eb85fd2a1"
      },
      "cell_type": "code",
      "source": "cols=[\"target\",\"ID_code\"]\nX = train.drop(cols,axis=1)\ny = train[\"target\"]\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "b79b286d0b1d7bd46525c73efb4690a7145c5cc1"
      },
      "cell_type": "code",
      "source": "X_test  = test.drop(\"ID_code\",axis=1)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "acaa4538c419e535e7ef5a68418047215cf8bd33"
      },
      "cell_type": "markdown",
      "source": "### Create  a sample model to calculate which feature are more important."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "56c4f79c0c33bc8710ee1fd30df191c4b7030d9f"
      },
      "cell_type": "code",
      "source": "train_X, val_X, train_y, val_y = train_test_split(X, y, random_state=1)\nrfc_model = RandomForestClassifier(random_state=0).fit(train_X, train_y)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "40f4fa49e4294d640b6f96aa27c4730f4be126c2"
      },
      "cell_type": "markdown",
      "source": " <a id=\"52\"></a> <br>\n## 5-2 How to calculate and show importances?"
    },
    {
      "metadata": {
        "_uuid": "a26af6d6608bb4102139f13863433e26c9402319"
      },
      "cell_type": "markdown",
      "source": "### Here is how to calculate and show importances with the [eli5](https://eli5.readthedocs.io/en/latest/) library:"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "9adf93667cd114f0c48b9d4bc362253a4a7ef6ad"
      },
      "cell_type": "code",
      "source": "import eli5\nfrom eli5.sklearn import PermutationImportance\n\nperm = PermutationImportance(rfc_model, random_state=1).fit(val_X, val_y)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "c785b21a2428bbe476539801876047f1250cc47a"
      },
      "cell_type": "code",
      "source": "eli5.show_weights(perm, feature_names = val_X.columns.tolist(), top=150)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "0b8ee511449fc9646b577d05ac84b3cacf4bb786"
      },
      "cell_type": "markdown",
      "source": "<a id=\"53\"></a> <br>\n## 5-3 What can be inferred from the above?\n1. As you move down the top of the graph, the importance of the feature decreases.\n1. The features that are shown in green indicate that they have a positive impact on our prediction\n1. The features that are shown in white indicate that they have no effect on our prediction\n1. The features shown in red indicate that they have a negative impact on our prediction\n1.  The most important feature was **Var_110**."
    },
    {
      "metadata": {
        "_uuid": "8861e65de7022f0aa5e091eddeff97ed6df1b76b"
      },
      "cell_type": "markdown",
      "source": "1. "
    },
    {
      "metadata": {
        "_uuid": "07ccdd2461ce8a4c3bbc3f349dfbaa2ee499890c"
      },
      "cell_type": "markdown",
      "source": "<a id=\"54\"></a> <br>\n## 5-4 Partial Dependence Plots\nWhile **feature importance** shows what **variables** most affect predictions, **partial dependence** plots show how a feature affects predictions.[6][7]\nand partial dependence plots are calculated after a model has been fit. [partial-plots](https://www.kaggle.com/dansbecker/partial-plots)"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "ee0955bd131d648bd537f2539bfb16109775ac7b"
      },
      "cell_type": "code",
      "source": "train_X, val_X, train_y, val_y = train_test_split(X, y, random_state=1)\ntree_model = DecisionTreeClassifier(random_state=0, max_depth=5, min_samples_split=5).fit(train_X, train_y)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "4e41acfbf4aac345477843189332b0244985b8a6"
      },
      "cell_type": "markdown",
      "source": "For the sake of explanation, I use a Decision Tree which you can see below."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "9883467c298aff26fa9067a3c52e81914b7613a8"
      },
      "cell_type": "code",
      "source": "features = [c for c in train.columns if c not in ['ID_code', 'target']]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "90b880a7ee20e9f9b4d730a6aa3533e9a613164a"
      },
      "cell_type": "code",
      "source": "from sklearn import tree\nimport graphviz\ntree_graph = tree.export_graphviz(tree_model, out_file=None, feature_names=features)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "3f0e5127ef1e07176ec276b54b14f71527b18e94"
      },
      "cell_type": "code",
      "source": "graphviz.Source(tree_graph)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "aa18ba89df82369acf1eeccc9ade46d89e9c65a3"
      },
      "cell_type": "markdown",
      "source": "As guidance to read the tree:\n\n1. Leaves with children show their splitting criterion on the top\n1. The pair of values at the bottom show the count of True values and False values for the target respectively, of data points in that node of the tree.\n><font color=\"red\"><b>Note: </b></font>\nYes **Var_81** are more effective on our model."
    },
    {
      "metadata": {
        "_uuid": "780dad7a7ba8563e8c3350bb12dbc3adddbff032"
      },
      "cell_type": "markdown",
      "source": "<a id=\"55\"></a> <br>\n## 5-5  Partial Dependence Plot\nIn this section, we see the impact of the main variables discovered in the previous sections by using the [pdpbox](https://pdpbox.readthedocs.io/en/latest/)."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "a2afa48cb3e7ed89e07b7ea5f14c67fa565c4a07"
      },
      "cell_type": "code",
      "source": "from matplotlib import pyplot as plt\nfrom pdpbox import pdp, get_dataset, info_plots\n\n# Create the data that we will plot\npdp_goals = pdp.pdp_isolate(model=tree_model, dataset=val_X, model_features=features, feature='var_81')\n\n# plot it\npdp.pdp_plot(pdp_goals, 'var_81')\nplt.show()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "076a43ef130a0c71834a783a450e3d1f7c34ff90"
      },
      "cell_type": "markdown",
      "source": "<a id=\"56\"></a> <br>\n## 5-6 Chart analysis\n1. The y axis is interpreted as change in the prediction from what it would be predicted at the baseline or leftmost value.\n1. A blue shaded area indicates level of confidence"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "cb28fd99a6b2bec709a8dda9af6b958cc28c75c1"
      },
      "cell_type": "code",
      "source": "# Create the data that we will plot\npdp_goals = pdp.pdp_isolate(model=tree_model, dataset=val_X, model_features=features, feature='var_82')\n\n# plot it\npdp.pdp_plot(pdp_goals, 'var_82')\nplt.show()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "e16c5a3a1301ddb137feb8010e9b48adaf4a2ba8"
      },
      "cell_type": "code",
      "source": "# Create the data that we will plot\npdp_goals = pdp.pdp_isolate(model=tree_model, dataset=val_X, model_features=features, feature='var_139')\n\n# plot it\npdp.pdp_plot(pdp_goals, 'var_139')\nplt.show()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "d051fc154bf318035daa976307774ddf3afe5135"
      },
      "cell_type": "code",
      "source": "# Create the data that we will plot\npdp_goals = pdp.pdp_isolate(model=tree_model, dataset=val_X, model_features=features, feature='var_110')\n\n# plot it\npdp.pdp_plot(pdp_goals, 'var_110')\nplt.show()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "5cf96fc2a60f0e0e80ab072d72d4c8677d027d3e"
      },
      "cell_type": "markdown",
      "source": "<a id=\"57\"></a> <br>\n## 5-7 SHAP Values\n**SHAP** (SHapley Additive exPlanations) is a unified approach to explain the output of **any machine learning model**. SHAP connects game theory with local explanations, uniting several previous methods [1-7] and representing the only possible consistent and locally accurate additive feature attribution method based on expectations (see the SHAP NIPS paper for details).\n\n<img src='https://raw.githubusercontent.com/slundberg/shap/master/docs/artwork/shap_diagram.png' width=400 height=400>\n[image credits](https://github.com/slundberg/shap)\n><font color=\"red\"><b>Note: </b></font>\nShap can answer to this qeustion : **how the model works for an individual prediction?**"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "57a5e96a5533e4f93d2eac8abd74b0225f9d3d81"
      },
      "cell_type": "code",
      "source": "row_to_show = 5\ndata_for_prediction = val_X.iloc[row_to_show]  # use 1 row of data here. Could use multiple rows if desired\ndata_for_prediction_array = data_for_prediction.values.reshape(1, -1)\n\n\nrfc_model.predict_proba(data_for_prediction_array);",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "4bd5eb8b532c72167ba192b94bfdffa5c5028fa9"
      },
      "cell_type": "code",
      "source": "import shap  # package used to calculate Shap values\n\n# Create object that can calculate shap values\nexplainer = shap.TreeExplainer(rfc_model)\n\n# Calculate Shap values\nshap_values = explainer.shap_values(data_for_prediction)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "f0870d8abb31c03d14ac611c518d206bfbe06351"
      },
      "cell_type": "markdown",
      "source": "If you look carefully at the code where we created the SHAP values, you'll notice we reference Trees in  **shap.TreeExplainer(my_model)**. But the SHAP package has explainers for every type of model.\n\n1. shap.DeepExplainer works with Deep Learning models.\n1. shap.KernelExplainer works with all models, though it is slower than other Explainers and it offers an approximation rather than exact Shap values."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "6b0f8b3fbc92ce157d40ca9de82eef16e61b2b02"
      },
      "cell_type": "code",
      "source": "shap.initjs()\nshap.force_plot(explainer.expected_value[1], shap_values[1], data_for_prediction)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "3d2c1aa7a32f36a4fc0e66f3aae135da63586269"
      },
      "cell_type": "code",
      "source": "# Calculate Shap values\nshap_values = explainer.shap_values(val_X)\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "bd284df5d62d611a529251e5a9e9ded42956f450"
      },
      "cell_type": "markdown",
      "source": " <a id=\"6\"></a> <br>\n# 6- Model Development\nSo far, we have used two  models, and at this point we add another model and we'll be expanding it soon.\nin this section you will see following model:\n1. lightgbm\n1. RandomForestClassifier\n1. DecisionTreeClassifier\n1. CatBoostClassifier"
    },
    {
      "metadata": {
        "_uuid": "d29905d41994b8e97236023a2339ca17dbc54018"
      },
      "cell_type": "markdown",
      "source": "## 6-1 lightgbm"
    },
    {
      "metadata": {
        "_kg_hide-input": true,
        "trusted": true,
        "_uuid": "e69efb809c800429126a0652ea2ad6d142a7748c"
      },
      "cell_type": "code",
      "source": "# based on following kernel https://www.kaggle.com/dromosys/sctp-working-lgb\nparams = {'num_leaves': 9,\n         'min_data_in_leaf': 42,\n         'objective': 'binary',\n         'max_depth': 16,\n         'learning_rate': 0.0123,\n         'boosting': 'gbdt',\n         'bagging_freq': 5,\n         'bagging_fraction': 0.8,\n         'feature_fraction': 0.8201,\n         'bagging_seed': 11,\n         'reg_alpha': 1.728910519108444,\n         'reg_lambda': 4.9847051755586085,\n         'random_state': 42,\n         'metric': 'auc',\n         'verbosity': -1,\n         'subsample': 0.81,\n         'min_gain_to_split': 0.01077313523861969,\n         'min_child_weight': 19.428902804238373,\n         'num_threads': 4}",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_kg_hide-input": true,
        "_kg_hide-output": true,
        "trusted": true,
        "_uuid": "ce482d4a37fbcaa01ddcfb181c5d81fac729a1e5"
      },
      "cell_type": "code",
      "source": "%%time\ny_pred_lgb = np.zeros(len(X_test))\nfor fold_n, (train_index, valid_index) in enumerate(folds.split(X,y)):\n    print('Fold', fold_n, 'started at', time.ctime())\n    X_train, X_valid = X.iloc[train_index], X.iloc[valid_index]\n    y_train, y_valid = y.iloc[train_index], y.iloc[valid_index]\n    \n    train_data = lgb.Dataset(X_train, label=y_train)\n    valid_data = lgb.Dataset(X_valid, label=y_valid)\n        \n    lgb_model = lgb.train(params,train_data,num_boost_round=2000,#change 20 to 2000\n                    valid_sets = [train_data, valid_data],verbose_eval=300,early_stopping_rounds = 200)##change 10 to 200\n            \n    y_pred_lgb += lgb_model.predict(X_test, num_iteration=lgb_model.best_iteration)/5",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "7f44325cc85d5a2b31fa8b8034a82630e3414bf7"
      },
      "cell_type": "markdown",
      "source": " <a id=\"62\"></a> <br>\n## 6-2 RandomForestClassifier"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "7ac7d44b0a6664339e8d4a58abacf35de4693163"
      },
      "cell_type": "code",
      "source": "y_pred_rfc = rfc_model.predict(X_test)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "889fbdd4dd0550e74a051079113b8042da7cf5a8"
      },
      "cell_type": "markdown",
      "source": " <a id=\"63\"></a> <br>\n## 6-3 DecisionTreeClassifier"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "b973d4bedc26288c23eba15c349e1e5cfe6631b5"
      },
      "cell_type": "code",
      "source": "y_pred_tree = tree_model.predict(X_test)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "3b95767d219b083d8e88341a291dfb01f4b328eb"
      },
      "cell_type": "markdown",
      "source": " <a id=\"64\"></a> <br>\n## 6-4 CatBoostClassifier"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "f743b2763fdef90213a713e0c2155c5232f799c8"
      },
      "cell_type": "code",
      "source": "train_pool = Pool(train_X,train_y)\ncat_model = CatBoostClassifier(\n                               iterations=3000,# change 25 to 3000 to get best performance \n                               learning_rate=0.03,\n                               objective=\"Logloss\",\n                               eval_metric='AUC',\n                              )\ncat_model.fit(train_X,train_y,silent=True)\ny_pred_cat = cat_model.predict(X_test)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "b576919e7514d144a1a34933d98c341bdc93a854"
      },
      "cell_type": "markdown",
      "source": "Now you can change your model and submit the results of other models."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "c2765114557a8a202652b6a0397f6c5f23217ca2"
      },
      "cell_type": "code",
      "source": "submission_rfc = pd.DataFrame({\n        \"ID_code\": test[\"ID_code\"],\n        \"target\": y_pred_rfc\n    })\nsubmission_rfc.to_csv('submission_rfc.csv', index=False)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "203de1a68091ba5b2f3281ba0a7ea508cfd34d81"
      },
      "cell_type": "code",
      "source": "submission_tree = pd.DataFrame({\n        \"ID_code\": test[\"ID_code\"],\n        \"target\": y_pred_tree\n    })\nsubmission_tree.to_csv('submission_tree.csv', index=False)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "40990b380a27e97f70caacbf81c6852a3842f0da"
      },
      "cell_type": "code",
      "source": "submission_cat = pd.DataFrame({\n        \"ID_code\": test[\"ID_code\"],\n        \"target\": y_pred_cat\n    })\nsubmission_cat.to_csv('submission_cat.csv', index=False)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "40990b380a27e97f70caacbf81c6852a3842f0da"
      },
      "cell_type": "code",
      "source": "submission_lgb = pd.DataFrame({\n        \"ID_code\": test[\"ID_code\"],\n        \"target\": y_pred_lgb\n    })\nsubmission_lgb.to_csv('submission_lgb.csv', index=False)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "f07cdaa9c8022f6ded23b632f67d805e0c4f5b4b"
      },
      "cell_type": "markdown",
      "source": " <a id=\"65\"></a> <br>\n## 6-5 Funny Combine "
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "9f1eeb3307d3c660c68cec153273b7e14e43ed73"
      },
      "cell_type": "code",
      "source": "submission_rfc_cat = pd.DataFrame({\n        \"ID_code\": test[\"ID_code\"],\n        \"target\": (y_pred_rfc +y_pred_cat)/2\n    })\nsubmission_rfc_cat.to_csv('submission_rfc_cat.csv', index=False)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "6bf9c325149e074d8ffb35277b755d177d3a38de"
      },
      "cell_type": "code",
      "source": "submission_lgb_cat = pd.DataFrame({\n        \"ID_code\": test[\"ID_code\"],\n        \"target\": (y_pred_lgb +y_pred_cat)/2\n    })\nsubmission_lgb_cat.to_csv('submission_lgb_cat.csv', index=False)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "cbd3cd8487f073be60343b208935965979e9d5c2"
      },
      "cell_type": "code",
      "source": "submission_rfc_lgb = pd.DataFrame({\n        \"ID_code\": test[\"ID_code\"],\n        \"target\": (y_pred_rfc +y_pred_lgb)/2\n    })\nsubmission_rfc_lgb.to_csv('submission_rfc_lgb.csv', index=False)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "435fe67f1a833ad62045a29288c440c6952d315c"
      },
      "cell_type": "markdown",
      "source": "you can follow me on:\n> ###### [ GitHub](https://github.com/mjbahmani/)\n> ###### [Kaggle](https://www.kaggle.com/mjbahmani/)\n\n <b>I hope you find this kernel helpful and some <font color='red'>UPVOTES</font> would be very much appreciated.<b/>\n "
    },
    {
      "metadata": {
        "_uuid": "a4fcaac6e4e792e6313b496c578aeaf5329ec5a3"
      },
      "cell_type": "markdown",
      "source": " <a id=\"7\"></a> <br>\n# 7- References & credits\nThanks fo following kernels that help me to create this kernel."
    },
    {
      "metadata": {
        "_uuid": "f67456896c63e2834e461c68ec6a960b60789663"
      },
      "cell_type": "markdown",
      "source": "1. [https://www.kaggle.com/dansbecker/permutation-importance](https://www.kaggle.com/dansbecker/permutation-importance)\n1. [https://www.kaggle.com/dansbecker/partial-plots](https://www.kaggle.com/dansbecker/partial-plots)\n1. [https://www.kaggle.com/miklgr500/catboost-with-gridsearch-cv](https://www.kaggle.com/miklgr500/catboost-with-gridsearch-cv)\n1. [https://www.kaggle.com/dromosys/sctp-working-lgb](https://www.kaggle.com/dromosys/sctp-working-lgb)\n1. [https://www.kaggle.com/gpreda/santander-eda-and-prediction](https://www.kaggle.com/gpreda/santander-eda-and-prediction)\n1. [https://www.kaggle.com/dansbecker/permutation-importance](https://www.kaggle.com/dansbecker/permutation-importance)\n1. [https://www.kaggle.com/dansbecker/partial-plots](https://www.kaggle.com/dansbecker/partial-plots)\n1. [https://www.kaggle.com/dansbecker/shap-values](https://www.kaggle.com/dansbecker/shap-values)\n1. [algorithm-choice](https://docs.microsoft.com/en-us/azure/machine-learning/studio/algorithm-choice)"
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.6.6",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 1
}